Sparse array cumsum leads to infinite recursion 62669 #62703
Conversation
pandas/core/arrays/sparse/array.py
Outdated
| if not self._null_fill_value: | ||
| if isinstance(self, SparseArray) and self.fill_value == 0: | ||
| # special case where we can avoid max recursion depth | ||
| return SparseArray(self.to_dense().cumsum()) |
There was a problem hiding this comment.
| if not self._null_fill_value: | |
| if isinstance(self, SparseArray) and self.fill_value == 0: | |
| # special case where we can avoid max recursion depth | |
| return SparseArray(self.to_dense().cumsum()) | |
| if not self._null_fill_value and self.fill_value != 0: |
There was a problem hiding this comment.
I see now this was wrong, but the previous implementation also did not handle the case
arr = pd.arrays.SparseArray([1.0, 0.0, np.nan, 3.0], fill_value=0.0)correctly as it would have given the result [1.0, 1.0, nan, nan] whereas I believe docstring indicates the result should be [1.0, 1.0, nan, 4.0]. We can accomplish this by:
if not self._null_fill_value:
return SparseArray(self.to_dense(), fill_value=np.nan).cumsum()Note this changes the fill_value to NaN, but that adheres to the docstring. Anytime this method has been successful in previous versions, it was always returning an NA fill value regardless of the input fill value. So I think we should stick to that logic at least for now.
Co-authored-by: Richard Shadrach <45562402+rhshadrach@users.noreply.github.com>
pandas/core/arrays/sparse/array.py
Outdated
| if not self._null_fill_value: | ||
| if isinstance(self, SparseArray) and self.fill_value == 0: | ||
| # special case where we can avoid max recursion depth | ||
| return SparseArray(self.to_dense().cumsum()) |
There was a problem hiding this comment.
I see now this was wrong, but the previous implementation also did not handle the case
arr = pd.arrays.SparseArray([1.0, 0.0, np.nan, 3.0], fill_value=0.0)correctly as it would have given the result [1.0, 1.0, nan, nan] whereas I believe docstring indicates the result should be [1.0, 1.0, nan, 4.0]. We can accomplish this by:
if not self._null_fill_value:
return SparseArray(self.to_dense(), fill_value=np.nan).cumsum()Note this changes the fill_value to NaN, but that adheres to the docstring. Anytime this method has been successful in previous versions, it was always returning an NA fill value regardless of the input fill value. So I think we should stick to that logic at least for now.
4 participants
doc/source/whatsnew/v3.0.0.rstfile if fixing a bug or adding a new feature.